BUG: make a failing Monte Carlo worker say so - #1182

__sim_producer binds sim_idx and inputs_json inside the loop, and its
handler reads both. A worker that fails in the seeding above the loop,
or in the claim that opens it, reached the handler with neither name
assigned and died with UnboundLocalError instead of recording anything.

The parent learns a worker failed from error_event, which the handler
sets on its last line, so it was never reached either and the run
waited rather than stopping.

Both names are bound before the try now, and the message says worker
startup when no index was claimed rather than naming one that does not
exist. Binding only the name in the traceback is not enough: the
message then raises on the other one, which the tests cover.

Scoped to the parallel producer. The serial handler has the same
unbound inputs_json and is RocketPy-Team#1177's.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

The workers say they failed by setting an event, and the parent joins
them and reads it. A worker that is killed runs no handler, so the
event stays clear, the join returns because the process is gone, and
the run reports the simulations it never wrote as done.

Measured with a worker leaving in the second simulation of six, two
workers: simulate() returned normally with two rows on disk.

Its exit code is what is left of a worker that ends this way, so the
parent reads that too. Anything other than zero is refused, None
included, since that is a worker that has not finished at all.

The handler around it holds the manager mutex while it reports, so a
failure in the reporting left the lock held by a process that had
already gone and the next worker waited on it. The event is set first
and outside the lock, the lock is released from a finally, and each
reporting step is separate so an unwritable log cannot replace the
failure being reported. A startup failure writes a row of its own now
rather than nothing, since the caller is told to read that file.

The test leaves through the data collector rather than a patched
method, since a spawn platform re-imports the module in the child and
never sees the patch, and through os._exit rather than a signal, since
SIGKILL is POSIX-only. Checked on both start methods. The Monte Carlo
objects are built on tmp_path rather than retargeted, because filename
is a plain attribute and the three log paths are set in __init__.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

The lock the workers share belongs to the manager and is not released
when the process holding it is killed. A sibling then blocks on a lock
nobody owns, and the parent, joining without a timeout, waits with it.
The exit code check the previous commit added is never reached, so the
one case it exists for is the one it cannot see.

The join polls now, and acts only when a worker has actually ended
badly. A run that is merely slow is never bounded: an exit code, not a
duration, is what says a worker is gone. The survivors are asked
through the event first, since one between simulations leaves with its
logs intact, and only the ones still running after that are ended.

Undoing this leaves every test in the new file red. Deciding on how
long a worker has taken instead of on how it ended leaves exactly one
red, which is the test that says a slow run must be left alone.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

Lifted out of __sim_producer unchanged. The producer was over pylint's
statement limit once the per-index seeding shortens it elsewhere, and
the handler is one thing rather than part of the loop around it.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

Three ways a failed run still passed for a finished one, all of them
found by review of the previous commits here.

The join waited only on exit status. A worker that fails the ordinary
way is caught by the producer, reports through the event and returns,
so it exits cleanly. With a sibling stuck, the parent saw one clean
exit and one live process and never stopped. A reported failure ends
the wait now as well, and a manager that cannot be asked is not taken
as evidence either way.

The bounded shutdown was undone one level up: after the exit-code
check raised, the outer handler joined every process again with no
timeout, so the stubborn worker it exists for was waited on anyway.
That handler uses the same bounded teardown now, and a test walks the
parallel path's syntax to keep an unbounded join out of it.

An exit code says how a process ended, never whether the index it had
claimed reached the logs, and the monitor counts claims rather than
rows. Measured, six simulations across two workers leaving through
os._exit(0): simulate() returned normally with nothing written. The
run is checked against the logs themselves at the end now. Both must
hold exactly the simulations asked for, none twice, none unreadable,
none numbered past the run.

The stand-in worker in the join tests gives up after two hundred
polls. Reproducing a real hang there would take a CI job down with it
rather than report, and requirements-tests.txt has no pytest-timeout.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

Review of the previous commits found three things, and reviewing my own
answer to the first of them found a fourth.

A worker kept sim_idx and inputs_json after committing a row, so a
failure in the claim that opened the next round was reported against
the simulation that had just succeeded, and that simulation's inputs
were written to the error log as well. Measured: output held index 0
and the error log held index 0 with the same inputs. They are cleared
once a row is committed, since nothing is in flight between two
simulations.

A worker whose event could not be set returned anyway, so it exited
cleanly and the parent saw neither an event nor a bad exit. It re-
raises when it could not announce, leaving the exit code to say so.

The bounded shutdown deserved more than one stage. It shares one
deadline across the fleet rather than giving each worker its own, uses
a monotonic clock so a correction cannot move a shutdown already under
way, and kills what outlives terminate.

The fourth is mine. I had made a reported failure end the wait, which
sounded right and was not: a worker that merely reports leaves nothing
behind, and its siblings stop of their own accord once they finish the
simulation in hand. Measured, a sibling that needed forty polls was
terminated after three. Only a worker that died can hold the shared
lock for good, so only that ends the wait now.

The completeness check no longer refuses rows numbered past the target.
An append given a smaller target than the checkpoint holds is an append
question, and refusing it here both changed what append means and said
rows were missing when they were extra.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

Codecov found the branch: a blank line is skipped rather than counted
as a row that cannot be read, and nothing exercised it. An interrupted
write is exactly what leaves them, so reporting a damaged log there
would fail a run that lost nothing.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

A collector's values are merged over the output record after the index
has been written into it, and the merge lets the right-hand side win,
so a key called index replaced the number of the simulation the row
belonged to. Measured on a two-simulation run with a collector
returning 999: both rows were written as 999.

Every check on a finished run reads that field, num_of_loaded_sims
counts on it, and an import keys results by it, so a run with such a
collector was writing a log nothing downstream could read correctly.
It was quiet until the completeness check turned it into a refusal
that named the wrong cause.

Refused where the collector is handed over, alongside the existing
export_list clash, so it costs nothing and happens before a file is
opened. A collector under any other name is untouched, which has a
test of its own.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>